Margin-based Feature Selection Techniques for Support Vector Machine Classification

Feature selection for classification working in high-dimensional feature spaces can improve generalization accuracy, reduce classifier complexity, and is also useful for identifying the important feature “markers”, e.g., biomarkers in a bioinformatics or biomedical context. For support vector machine (SVM) classification, a widely used feature selection technique is recursive feature elimination (RFE). In recent work, we demonstrated that the RFE objective is not generally consistent with the margin maximization objective that is central to the SVM learning approach. We thus proposed explicit margin-based feature elimination (MFE) for SVMs and demonstrated both improved margin and improved generalization accuracy, compared with RFE for the case of linear SVMs. In this paper, after reviewing MFE, we first introduce an extension which achieves further gains in margin at small computational cost. This extension solves the SVM optimization problem to maximize the classifier’s margin at each feature elimination step, albeit in a lightweight fashion by optimizing only two degrees of freedom – the weight vector’s slope and intercept. We next consider the case of a nonlinear kernel. We show that RFE defined for the nonlinear kernel case assumes that the weight vector length is strictly decreasing as features are eliminated. We demonstrate experimentally that this assumption is not in general valid for the Gaussian kernel and that, consequently, RFE may give poor results in this case. An extension of MFE for the nonlinear kernel case gives both better margin and generalization accuracy. This approach may help nonlinear kernel SVMs to avoid overfitting and, thus, to achieve better results than linear SVMs in some high-dimensional domains where use of nonlinear kernels has not to date been found very favorable.